Skip to content

ipparse: LuaJIT pack perf, Lua 5.1/5.4/5.5 compat, Lunatik v5.1 fixes, kernel test suite and CI - #2

Merged
jperon merged 43 commits into
masterfrom
ipparse-perf-compat
Nov 10, 2026
Merged

jperon merged 43 commits into
masterfrom
ipparse-perf-compat

Conversation

@jperon

@jperon jperon commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

ipparse performance work for LuaJIT, Lua 5.1/5.4/5.5 compatibility, fixes for real Lunatik usage (upstream v5.1, Lua 5.5 in the kernel), a kernel-side portable test suite, VM harness fixes and a multi-runtime CI.

Performance (LuaJIT)

  • pack_compat_lib: formats are compiled once into byte writers that fill a shared buffer; single string.char; no varargs/closures in the hot loop; no tail-call returns (they aborted traces with NYI: return to lower frame); trailing-string fast path; reused argument array. No trace aborts left under -jv.
  • l7/dns: single concat in message pack, string.byte in label.
  • DNS response pack: ~45 µs → ~1.1 µs; LuaJIT is now ahead of every PUC Lua on both parse and pack.

Compatibility

  • Lua 5.3+: variadic band/bor/bxor in bit53 (TCP flags were dropped by tcp.new).
  • lib/sha.lua: three wrong SHA-256 constants, 2-arg XOR called with 3 args, HMAC double-feed (broke HKDF/QUIC keys on Lua 5.3+).
  • Lua 5.1: pure-Lua bitwise fallback (bit_pure), FFI made optional in pack_compat_lib (pure-Lua 64-bit ints and IEEE 754), \xNN test escapes.
  • LuaJIT: pack("<d", x) returned an empty string; negative 64-bit writes were undefined (double → uint64_t), wrong on older LuaJIT.

Lunatik (kernel)

  • ipparse.fun overflowed the C-stack guard when loaded one require level down under v5.1 (real usage, e.g. l7.quic.session): protected flattened. Margin: loading still fails at depth 5.
  • AEAD decrypt: handle both conventions (old: raises; v5.1: returns nil, "EBADMSG") and every errno-name form (EBADMSG, -EBADMSG, -74); a plaintext equal to "EBADMSG" is no longer rejected.
  • No floats/os in kernel paths: 8-byte QUIC varints, tcp_stream clock (injectable, linux.time fallback).
  • bidirectional reverse cache revalidated with rawget (stale entries after live updates).

Tests, harness, CI

  • New kernel suite test_lunatik_kernel_portable (parse/pack modules run inside the kernel).
  • vmtest: guest gets the resolved build paths, symlinked build dir ignored, per-run scratch dir.
  • Lunatik pinned to upstream tag v5.1 (commit checked); non-blocking kernel-lunatik-head job against master (or any ref via workflow_dispatch); VM console/output uploaded on failure.
  • Userspace matrix: LuaJIT, Lua 5.1, 5.4, 5.5 (all blocking).

Verification (local)

  • Every commit passes the LuaJIT suite (42/42).
  • Head: Lua 5.1 335/335, 5.4 360/360, 5.5 360/360, LuaJIT 335/335.
  • VM (kernel 6.18.35, Lunatik v5.1): kernel crypto green, kernel portable suite 296/296, All tests passed in VM.
  • actionlint clean.

🤖 Generated with Claude Code

jperon and others added 30 commits November 10, 2026 21:13
LuaJIT's bit.band/bor/bxor accept any number of arguments, but the
Lua 5.3+ compatibility layer only handled two. This broke e.g. TCP flag
construction (bor with seven arguments), failing 6 test_tcp cases under
Lua 5.4/5.5. Fold left to right, with a direct path for the two-argument
case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- sha256_K had three wrong constants (K[17], K[18], K[54]).
- The XOR helper was binary-only but Sum0/Sum1 call it with three
  terms, so the third term was silently dropped.
- hmac() fed the message twice before finalizing the inner hash; feed
  it once from the returned chunker instead. Also fixes the streaming
  path, where the message was finalized prematurely.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Save and restore package.preload/package.loaded entries for crypto,
crypto.ecb and the lunatik backend so the crypto stubs cannot leak into
modules loaded later in the same test run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ECB stub invented a crypto.ecb module with encryptblock(); Lunatik
only exposes require("crypto").skcipher (lib/luacrypto_core.c registers
"skcipher") whose objects provide encrypt(iv, data)
(lib/luacrypto_skcipher.c luacrypto_skcipher_encrypt). Stub skcipher and
use an empty IV, matching ECB ivsize == 0 in luacrypto_checkiv.

Non-auth errors: luacrypto_aead_finish() throws every failing crypt
return code via lunatik_throw (lib/luacrypto_aead.c:144), and a too-short
input throws -EBADMSG (line 186). The backend wraps decrypt in pcall and
maps EBADMSG to a nil,error authentication failure while other failures
become nil,"aead(gcm(aes)) decrypt failed: ...", consistent with the
documented nil,error backend contract. Expect nil,error in the test
instead of a propagated exception.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the per-argument switch dispatch and ffi buffers with a
per-format plan of small emitter closures built once and cached.
Integers up to 4 bytes are emitted with string.char arithmetic, so
LuaJIT no longer abandons traces; byte output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace per-section comprehension closures in pack with plain for loops
filling one parts table, concatenated once; the body length is accumulated
so header.size stays identical. Read label length and pointer bytes with
string.byte after need_bytes has already validated the offsets.

Bench (200k msgs, median of 4 interleaved runs, µs/msg):
LuaJIT parse 3.93 -> 2.94, pack 3.30 -> 2.96
Lua 5.4 parse 8.01 -> 7.04, pack 2.58 -> 1.87

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…able

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it bounds

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ield strings

Replace the per-field string encoders with writers that fill a module-level
byte buffer, then emit one string.char per run of integers. String options
(c, s, z) and floats flush the buffer first; arguments a1..a8 avoid select
and vararg in the fast path. This lets LuaJIT compile pack instead of
falling back to the interpreter (no FNEW/VARG aborts).

LuaJIT 2.1, us per call (before -> after):
  pack(">H H")                 0.259 -> 0.122
  pack(">H B B H H H H")       0.847 -> 0.187
  pack(">H H I4 s2")           0.664 -> 0.285
  full DNS message pack         3.814 -> 1.904

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guest re-sources lib.sh, which recomputed BUILD_DIR from
VMTEST_BUILD_DIR. That variable was never exported into cmd.sh, so a
cache outside the repo was silently replaced by the default path inside
the read-only repo, and guest-test.sh failed on kernelrelease.

Resolve BUILD_DIR and LUNATIK_SRC to absolute paths in lib.sh and export
both from the host into the guest wrapper.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A trailing-slash pattern only matches directories, so a natural
`tools/vmtest/build -> /shared/cache` symlink showed up as untracked.
Anchor the patterns with a leading slash, which also covers symlinks.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All invocations wrote cmd.sh/out/rc/console.log under the same fixed
/tmp/vmtest-rw. Two concurrent runs removed each other's directory and
one failed with "VM did not run the guest command".

Use mktemp -d per run (removed on success, kept on failure for the log),
while keeping VMTEST_RW_DIR as an explicit opt-in override.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Use 64-bit shifts/masks when values are native integers (Lua 5.3+ and the
Lunatik kernel, where math.floor and floats are unavailable); keep the
high/low division split on LuaJIT doubles. Byte output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y module

require ipparse.lib.pack_compat returns string.pack on Lua 5.3+ and the
fallback on LuaJIT, so the test module also parses under the Lunatik kernel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read the current time through a configurable clock returning an integer in
fixed units instead of os.time, which is absent in Lunatik. Defaults to
os.time (1 Hz) when available, then linux.time (1 GHz) when the kernel
module is loaded, and errors at creation when no clock exists.

purge(max_age) now compares against max_age * clock_hz, keeping the
comparison integer-only (no division, no floats).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mirror frames.moon's native_int check (math.floor == nil or math.type ~=
nil) so the maximum-value case also runs in the Lunatik kernel, where
math.type is absent but native 64-bit integers are available.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LuaJIT aborted the traces exiting pack's loop with "NYI: return to lower
frame" because `return table.concat(parts)` / `return buf_string(...)`
are tail calls, whose return target frame is not recordable. Assigning the
result to a local and returning it removes every abort for b4.lua.

LuaJIT 2.1, bench b4.lua (median of 5, µs):
  msg pack            1.76 (was 1.79-3.39, noisy)
  rr pack             0.28
  sp >H H I4 s2       0.24
  sp >H B B H H H H   0.19
  sp >H H             0.12
-jv: no abort/NYI/blacklist on pack_compat_lib.lua (was 3 kinds).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When every option but the last only writes integer bytes and the last one
emits a string (s/c/z/f/d as the final field), skip the parts list and
table.concat entirely: write the length prefix (if any), flush the byte
buffer, and concatenate once with the emitted string.

This covers pack_rr's ">H H I4 s2", EDNS's ">Hs2" and the zero/lib
string cases.

LuaJIT 2.1, bench b4.lua (median of 5, µs):
  msg pack            1.41 (was 1.76)
  rr pack             0.20 (was 0.28)
  sp >H H I4 s2       0.17 (was 0.24)
  sp >H B B H H H H   0.18 (was 0.19)
  sp >H H             0.12 (was 0.12)
-jv: still no abort/NYI/blacklist on pack_compat_lib.lua.
Tests: LuaJIT 296/296, Lua 5.4 309/309.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pack allocated a fresh 8-slot table on every call, which dominated small
formats. Reuse a module-level _args array instead (like _buf, pack never
yields or calls user callbacks, so it is not reentrant). Formats consuming
more than 8 values keep the one-off vararg table.

LuaJIT 2.1, bench b4.lua (median of 5, µs):
  msg pack            1.19 (was 1.41)
  rr pack             0.18 (was 0.20)
  sp >H H I4 s2       0.137 (was 0.17)
  sp >H B B H H H H   0.146 (was 0.18)
  sp >H H             0.088 (was 0.12)
-jv: still no abort/NYI/blacklist on pack_compat_lib.lua.
Tests: LuaJIT 296/296, Lua 5.4 309/309.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lunatik_pusherrname emits "EBADMSG" on kernels >= 6.7, and on older
kernels "%pE" of ERR_PTR(-err): "-EBADMSG" with CONFIG_SYMBOLIC_ERRNAME,
or the signed decimal "-74" otherwise. Match these exact whole strings
(after stripping Lua's optional "chunk:line: " prefix), so lookalike
errnos such as -174 are not mistaken for a tag mismatch.

decrypt always raises via lunatik_throw on failure and returns only the
plaintext on success, so drop the bogus plaintext == "EBADMSG" check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lua 5.1 (PUC) has no ffi module, so requiring it failed before any pack
operation could run:

  ./ipparse/lib/pack_compat_lib.lua:5: module 'ffi' not found

Load FFI with pcall and keep the LuaJIT fast paths unchanged. Without FFI:

- native endianness defaults to little-endian (no reliable probe);
- integers up to 8 bytes use double arithmetic (2^53 precision), with
  signed l/j/i8 decoded by byte-wise two's complement so small negatives
  such as -2 stay exact;
- f/d/n use a pure-Lua IEEE 754 codec handling zero, -0, infinities,
  NaN, normals, subnormals and round-to-nearest-even.

Also fix the FFI float/double byte extraction, which used a table
comprehension that produced an empty table (pack of a double returned
an empty string), and decode i8 through int64_t instead of the lossy
uint64 path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lua 5.1 does not support hexadecimal escapes in string literals, so test
data such as "\x01hello\xff" was read literally as x01helloxff. This
produced failures across most modules under Lua 5.1, for example:

  FAIL init: hex2bin converts hex to bytes
  FAIL l2/ethernet: mac2s converts binary to string
       expected 'aa:bb:cc:dd:ee:ff', got '78:61:61:78:62:62'
  FAIL l3/ip4: ip42s converts 4 bytes to dotted decimal
       expected '192.168.1.1', got '120.99.48.120'
  FAIL quic/varint: parse 1-byte value 0 ... got 14384

Rewrite every \xNN escape in the test sources as the equivalent \DDD
decimal escape, which is valid in Lua 5.1 through 5.5 and LuaJIT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jperon and others added 13 commits November 10, 2026 21:13
Track the official v5.1 release instead of a rewriting fork branch, and
verify the tag still resolves to the expected commit so a moved tag fails
loudly. An absent LUNATIK_SRC is cloned even when explicitly named; a stale
harness-managed cache is re-cloned, while a user-provided checkout is only
reported. Build code generation with the Lua 5.4 toolchain already used by
CI (the module itself embeds Lua 5.5).

Also key the kernel CI cache on KERNEL_VERSION and kernel.fragment only, so
changing the Lunatik pin does not trigger a needless kernel rebuild.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run the VM suite with a fixed VMTEST_RW_DIR and upload the guest's
console.log and out from it, instead of the useless host dmesg.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Install the interpreters through Nix (Lua 5.5 is not in apt on
ubuntu-latest), compile MoonScript once, then reuse the generated Lua files.
Lua 5.1 is marked continue-on-error until its support lands.

The Makefile gains LUA ?= luajit, used by test-nobuild and example, keeping
the default behaviour unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add kernel-lunatik-head, which runs the same VM suite against
luainkernel/lunatik at a workflow_dispatch-selected ref (default master),
with its own LUNATIK_SRC and artefact. LUNATIK_REF/LUNATIK_COMMIT are
overridable from the environment so versions.env only sets defaults.

Document the upgrade procedure at the top of the workflow. Provide libbpf
for master's eBPF loader, and make the guest install use the writable 5.4
Lua module paths instead of the read-only Nix store.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/luacrypto_aead.c in v5.1 replaces lunatik_throw(L, -EBADMSG) with
lunatik_pushfail(L, -EBADMSG), so a bad tag yields nil, "EBADMSG"
instead of raising. aes_128_gcm_decrypt only inspected the pcall status
slot and lost the error, returning nil, nil.

Check both slots: the pcall error (old API) and, when the plaintext is
nil, the second return value (v5.1+). Other failures still raise, as
luacrypto_aead_finish and luacrypto_skcipher_crypt keep using
lunatik_throw; encrypt and ECB therefore need no change.

Co-Authored-By: Claude Opus 5.5 <noreply\@anthropic.com>
bit_pure is only the last-resort fallback of bit_compat (Lua 5.1 without
`bit`) and relies on math.floor, which the Lua 5.5 kernel runtime does not
provide. Guard the bit_pure cases behind math.floor so the kernel run keeps
only the bit_compat cases instead of failing 16 assertions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Lua 5.5 kernel runtime has no floats: a single float literal makes the
whole chunk fail to compile ("malformed number near '.5'"), which also
dropped the integer cases of the module. Move every float/double/IEEE test
into ipparse/tests/lib/test_pack_compat_float.moon, registered in run_all
but deliberately absent from test_lunatik_kernel_portable. The remaining
module has no float literal, no `^` and no `/` in code, so it loads and runs
in the kernel.

The native kernel string.pack is correct for negative >i8: the pinned
Lunatik v5.1 lua/ sources built with -D_KERNEL (Lua 5.5.1, 64-bit
lua_Integer, SZINT=8) return fffffffffffffffe for pack(">i8", -2) and
round-trip -2; both cases PASS in the VM now that the module compiles.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lunatik v5.1 computes LUAI_MAXCCALLS from the kernel stack actually
consumed, and the Lua 5.5 parser is recursive in C. Loading ipparse.fun
through a second require level (e.g. script -> ipparse.l7.dns -> fun)
overflowed the 16 KiB kernel stack because `protected` nested two inline
function expressions inside the xpcall call.

Define the error handler and the returned function as block-level locals
and build the per-call closure as a statement inside it. Parser C
recursion drops enough that ipparse.fun now loads up to four require
levels deep (measured: depth 4 OK, depth 5 overflows), covering real
usage (script -> ipparse.l7.quic.session -> ipparse.l4.quic -> fun, or
script -> ipparse.l7.dns -> fun). API and semantics are unchanged: same
26 arguments/return values, same xpcall error handling, same optional
`op` callback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most modules exercised by the portable kernel suite require
ipparse.fun two or three `require` levels down, inside a pcall(require …)
chain that already consumes kernel stack. Load it once at the shallowest
C-stack point before the loop so the suite no longer depends on that
chain depth and stays clear of Lunatik's stack-derived C-call guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The double -> uint64_t conversion is undefined for negative values, and
older LuaJIT releases saturate it to 0x8000000000000000, corrupting 64-bit
signed writes. Route negatives through int64_t first, then reinterpret the
bits as unsigned, keeping the direct cast for non-negative values (so
values >= 2^63 are unaffected).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reverse-lookup cache kept returning a stale key after the forward
entry was modified or deleted. Validate the cached key with
rawget(self, key) == val before reuse; rawget avoids re-entering the
__index metamethod, which would recurse on an absent key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@jperon
jperon merged commit a77fd2f into master Nov 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant